feat: select a wake-up adapter automatically - #77

Closes #72.

Why

A commit in a web process did not wake a worker process unless somebody
configured an adapter, so delivery waited out the polling interval, up to the
one-second idle_polling_interval default. The adapters existed and were
measured. Nothing chose between them.

One setting, not two

My first draft added config.wake_up = :automatic beside the existing
config.wake_up_adapter. That was wrong. Rails uses one setting that takes
either a name or an object, never a parallel pair:
config.cache_store accepts
:redis_cache_store or a cache object, and config.active_job.queue_adapter
accepts :sidekiq or an adapter.

So config.wake_up_adapter takes both:

config.wake_up_adapter = :automatic     # default
config.wake_up_adapter = :in_process    # opt out
config.wake_up_adapter = :postgresql    # force one
config.wake_up_adapter = MyAdapter.new  # your own

An unknown name raises ArgumentError naming the accepted values, rather than
quietly polling. configure validates it, so a typo fails at boot rather than
at the first wake-up, which is after a commit. A typo that silently costs a
second of latency is the failure this change exists to remove.

A name that exists but that the environment cannot provide is different. It
polls, logs solid_objects.wake_up.unavailable once, and records the reason in
the capability, so the doctor warns. :postgresql on a database with no
notification channel used to build the adapter anyway and report
crosses_processes: true, so the doctor said PASS while every LISTEN failed.
:redis without SOLID_OBJECTS_REDIS_URL used to reach for the client default
rather than say it had no address.

Selection

1. A configured adapter or name wins.
2. SOLID_OBJECTS_REDIS_URL selects Redis, on any database.
3. PostgreSQL selects LISTEN, unless a probe notification fails to arrive.
4. Everything else polls.

The probe proves delivery

LISTEN does not survive a transaction pooler such as PgBouncer, so the path is
probed before it is chosen.

My first version set application_name and read it back on the next statement.
Review was right that this proves nothing. A pooler can hand the same backend to
two consecutive autocommit statements, so the read-back succeeds while LISTEN
still has no session affinity, and selection would memoise a notification path
that never fires.

Selection now proves the path end to end. It listens on a probe channel, sends
one NOTIFY from a second connection, and waits up to two seconds for it to
arrive.

def notifications_deliver?
  probe = Postgresql.new(channel: PROBE_CHANNEL)
  return false unless probe.listen
  return false unless notify_probe_channel

  probe.wait(timeout: PROBE_TIMEOUT_SECONDS)
rescue
  false
ensure
  probe&.stop
end

Three points about the shape:

• Backend reuse is no longer evidence. A pooler that reuses one backend still
  drops the asynchronous notification, because the client connection is not
  bound to the backend that ran LISTEN.
• The NOTIFY goes through a separate connection rather than the pooled one.
  NOTIFY delivers at commit, so a caller that selects inside an open
  transaction would otherwise never see its own probe.
• A probe that cannot run is no longer treated as support. Failure falls back to
  polling, which costs latency and not correctness. The probe closes both
  connections, so selection leaves nothing open.
• The probe has its own channel. On the production channel, every booting
  process would wake every waiting role in the deployment once.

Selection runs once per process

The probe found a second defect, in code this branch did not add.
SolidObjects.wake_up memoised with ||= and no lock. One selection was cheap,
so the race cost nothing visible. A probe that opens connections and waits for a
notification is not cheap. Eight threads that raced for the first use each ran a
full selection, took every free pool connection, and left four callers waiting
out the checkout timeout:

EnqueueTest#test_allocates_unique_sequences_under_concurrent_enqueue
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the
pool within 5.000 seconds (waited 5.012 seconds); all pooled connections were
in use

Selection now runs under a mutex, so one thread probes and the rest read the
result. A new test holds it: eight threads race, and selection must run once.
Without the lock it failed with Expected: 1, Actual: 8.

Reporting the choice

SolidObjects.wake_up.capability
# => #<data SolidObjects::WakeUpCapability
#      adapter=:postgresql_notify, crosses_processes=true,
#      measured_floor_ms=2.9,
#      reason="a probe notification arrived, so PostgreSQL LISTEN carries the signal between processes">

bin/rails solid_objects:doctor reports the same record: PASS when it crosses
processes, WARN with the reason when it does not.

This also fixed a bug the change would otherwise have introduced. The
polling-only warning guarded on return if configuration.wake_up_adapter, which
is now always truthy, so the warning would have stopped firing everywhere. It
guards on capability.crosses_processes instead, which is what it meant to ask.
An adapter that reports its own capability keeps it. A configured
SolidObjects::WakeUp signals in one process only and says so, so the doctor
warns and the process registry keeps its polling warning, rather than have
selection relabel it as cross-process. An adapter that reports no capability is
somebody else's and is still assumed to cross processes, as before.

Behaviour change

PostgreSQL deployments that configure nothing now use notifications. They
gain cross-process wake-up, and with it a connection per waiting thread outside
the pool and one NOTIFY per enqueue after the commit. The NOTIFY lands after
COMMIT, so it does not widen the lock window, but it is one more round trip
per enqueue. config.wake_up_adapter = :in_process keeps the old behaviour.

Tests

test/integration/wake_up_selection_test.rb is new, 22 tests: an explicit
adapter wins, a configured adapter keeps the capability it reports about itself,
the doctor warns about a configured in-process adapter, an adapter that reports
none is recorded as configured, racing threads select once, a requested
:postgresql polls when the database has no channel, a requested :redis
polls when no URL is set, a listener wakes from a notification sent on another
connection, :in_process opts out, a name selects without
probing, an unknown name is refused when the configuration is validated and again at
selection, a Redis URL wins on any database, PostgreSQL
selects notifications when a probe notification arrives, PostgreSQL polls when
one does not, a pooled session falls back and warns once, a database without a
channel reports its floor, the capability names the adapter actually installed,
an unreachable database degrades to in-process rather than raising, an adapter
without watch is accepted, one that cannot signal is refused, and the doctor
reports it.

configure validates the setting, and the check follows the contract the roles
actually use. Four roles ask respond_to?(:watch) before they call it, and
ADR 0011 records that an adapter without it keeps the fast cadence, so the check
requires signal and wait only.

Each of these failed first. The two that hold the review findings failed for the
stated reason:

WakeUpSelectionTest#test_a_configured_adapter_keeps_the_capability_it_reports_about_itself
Expected: :in_process
  Actual: :configured

WakeUpSelectionTest#test_postgresql_polls_when_a_probe_notification_does_not_arrive
Expected: :polling
  Actual: :postgresql_notify

test/unit/wake_up_adapters_test.rb now stubs the probe in both directions, so
family routing is asserted without depending on which database the suite runs
against.

Two existing tests moved to :in_process rather than changing what they assert.
The polling warning test is about the warning, not about selection. The enqueue
statement count is about the enqueue transaction, not about the NOTIFY a
cross-process adapter adds after it. That count guard is what caught the extra
statement in the first place.

Validation

bundle exec rake passes. Steep reports no type error, Brakeman no warning.

The positive PostgreSQL case is measured rather than assumed. The selection file
finishes in 0.93 s, so the probe notification arrived rather than ran out its
two-second timeout.

What this does not do

The issue's third part asks for a measurement before choosing a MySQL notifier,
between a tighter poll floor, a signal table, and GET_LOCK. No MySQL
notifier is shipped here, so no measurement gates this change: MySQL still
polls, and the capability now says so and reports its floor rather than leaving
an operator to infer it. Choosing one of those three still needs the benchmark
the issue describes, and belongs in its own change.